1.1 Background

The effect of default priors on BMA, model selection, and predictive performance have been studied in range of disciplines including, econometrics, social sciences, biostatistics. Ley and Steel (Ley and Steel 2008) evaluated the predictive performance of several priors including Maruyama-George (Maruyama and George 2011), Bottolo-Richardson (Bottolo and Richardson 2010), Hyper g, Hyper g/n, and Zellner Siow (A Zellner and Siow 1980) priors. Combining Binomial-Beta priors on model sizes with g-priors on the coefficients of each model, they proposed a benchmark Beta prior as well as a hyper-g/n prior for econometric applications, specifically cross-country growth regression. Eicher, Papageorgiou, and Raftery (Eicher, Papageorgiou, and Raftery 2011) evaluated 12 priors which have been proposed in statistics and economics literature and found that the Unit Information Prior (UIP), which corresponds to the Bayesian Information Criteria (BIC) approximation of the marginal likelihood, combined with the uniform prior over the model space, outperformed, the 11 other priors. Liang et al. assessed the predictive performance of a range of g-priors and mixtures of g-priors using the highest probability models under each prior, rather than BMA (Liang et al. 2008). Based upon reported mean squared error (MSE) of the predictions, they concluded that there were no statistically significant differences in the predictive performance of the mixtures vis-a-vis the fixed g priors. Fernandez et al. (Fernah Ndez, Ley, and Steel 2001) evaluated nine priors based upon Zellner’s g-prior structure (Arnold Zellner and Moulton 1985) and recommended a g-prior equal to \(1/k^2\) when \(n\leq k^2\) and \(1/n\) when \(n > k^2\), where \(n\) is the number of observations and \(k\) is the number of regressors.

The papers is organized as follows. Section 2 summarizes BMA key concepts, with a focus on prior parameter specification. Section 3 introduces the motivating example and the data used for this experiment. Section 4 includes a univariate and bivariate exploratory data analysis (EDA). Section 5 covers model prior comparison, evaluation and selection. Section 6 reports the prediction accuracy of top performing models on unseen data. Finally, section 7 includes concluding remarks.

BMA modeling and prediction functionality was provided by the BAS package (M. Clyde 2017). Core scripts and functions used in this study are included in the appendix. The complete source code used to produce this report is also available on github at https://github.com/DataScienceSalon/Bayesian-Regression.

---

2.1 Bayesian Model Averaging (BMA)

The objective of variable selection in linear regression is to identify a set of candidate predictors that produce the “best” model. Here, “best” model may mean the model that best predicts unseen cases, or that which is best explained by the data. Given an independent variable \(Y\) and a set of \(k\) candidate predictors \(X_1, X_2, ..., X_k\), the “best” model is more concretely described as follows: \[Y = \beta_0 + \displaystyle\sum_{j=1}^{p}\beta_jX_j + \epsilon = X\beta + \epsilon\] where:
\(X_1, X_2, ..., X_p\) is a subset of \(X_1, X_2, ..., X_k\)
\(X\) is a \(n \times (p + 1)\) matrix containing the observed data on \(p\) predictors
\(\epsilon\) ~ \(N(0,\sigma^2I)\)
\(\beta\) and \(\sigma^2\) are the \((p + 1)\) individual unknown parameters

However, model selection exercises that lead to a single “best” model ignore model uncertainty (Hodges 1987; Draper 1995; Raftery, Madigan, and Hoeting 1997), and leads to underestimation of uncertainty when making inferences about quantities of interest (Raftery, Madigan, and Hoeting 1997).

Bayesian model averaging, employed when a variety of statistically reasonable models are extant, addresses model uncertainty and leads to optimal predictive ability (A. Raftery, Madigan, and Hoeting 1994) by average over all possible combinations of predictors when making inferences about quantities of interest. The final estimates are computed as a weighted average of the parameter estimates from each of the models.

The standard BMA solution, first introduced by Leamer in 1978, defines a set of all possible models as \(M = ({M_1,..., M_k})\). If \(\Delta\) is the quantity of interest, such as a movie prediction or tomorrow’s stock price, then the posterior distribution of \(\Delta\) given the data \(D\) is: \[Pr(\Delta|D) = \displaystyle\sum_{k=1}^KPr(\Delta|M_k, D)Pr(M_k|D)\] This is an average of the posterior distributions under each model weighted by the corresponding posterior model probabilities (Raftery, Madigan, and Hoeting 1997). The posterior model probability of \(M_k\) is then computed as the ratio of its marginal likelihood to the sum of the marginal likelihoods over the entire model space and is given by(Amini and Parmeter, n.d.):

\[Pr(M_k|D) = \frac{Pr(D|M_k)Pr(M_k)}{\displaystyle\sum_{i=1}^{K}Pr(D|M_i)Pr(M_i)}\] where:
\[Pr(D|M_k) = \int p(D|\theta_k, M_k)P(\theta_k|M_k)d\theta_k\] is the marginal likelihood of model \(M_k\), \(\theta_k\) is the vector of parameters of model \(M_k\), \(Pr(\theta_k|M_k)\) is the prior density of \(\theta_k\) under model \(M_k\), \(Pr(D|\theta_k, M_k)\) is the likelihood of the data given the \(M_k\) and \(\theta_k\), and \(Pr(M_k)\) is the prior probability that \(M_k\) is the true model.

At this stage, the posterior inclusion probability of each candidate predictor \(\beta_p\), \(Pr(\beta_p\neq0|D)\), is obtained by summing the posterior model probabilities over all models that include \(\beta_p\). Referring back to the linear regression model; the posterior means and standard deviations of coefficient vectors \(\beta\), are defined as: \[E[\hat{\beta}|D] = \displaystyle\sum_{j=1}^{2^k}\hat{\beta}Pr(M_j)|D),\] \[V[\hat{\beta}|D] = \displaystyle\sum_{j=1}^{2^k}Var[{\beta|D,M_j] +\hat\beta^2})Pr(M_j|D) - E[\beta|D]^2.\] Averaging over all models in this way provides better predictive results, as measured by the logarithmic scoring rule, than any single model \(M_j\) \((j 1,...,K)\) (Raftery, Madigan, and Hoeting 1997).

2.2 Prior Distributions of Parameters

To implement BMA, one must specify prior distributions over all parameters in all models, as well as prior probabilities of the models themselves. If prior information about the parameters and the models is available, it should be used. However, if the amount of prior information is small relative to the effort required to specify it, as is often the case, default or so-called “non-informative” or “reference” priors may be used for such analysis. The selection of default priors may effect the integrated likelihood, the key factor in computing posterior model weights, and so the prior density should be wide enough and reasonably flat over the region of the parameter space where the likelihood is large, but not so spread out as to decrease the prior at the posterior mode. This decreases the integrated likelihood and may unnecessarily penalize larger models (A. Raftery, Madigan, and Hoeting 1994).

This experiment explores the following nine default priors supported in literature (Table 1).

Table 1: Parameter priors

#	Prior	Comment	Source
1	BIC	Reference prior based upon log marginal likelihood estimated using Bayesian information criterion.	Schwarz (1978)
2	AIC	Reference prior based upon log marginal likelihood estimated using Akaike information criterion.	Akaike (1973)
3	Empirical Bayes (Global) Prior	Empirical bayes method using a global estimate of g from the marginal likelihood of g.	George and Foster (2000) and Clyde and George (2000)
4	Empirical Bayes (Local) Prior	Empirical bayes method using a model specific estimates of g.	George and Foster (2000) and Clyde and George (2000)
5	g-prior	Informative, conjugate normal prior conditioned on variance with prior mean and data dependent variance/covariance.	Zellner (1986)
6	Hyper-g	Class of g-priors based upon continuous proper hyperprior f(g), giving a mixture of g-priors where the prior on g/(1+g) is a Beta(1, alpha/2).	Liang, Paulo, Molina, Clyde and Berger (2007)
7	Hyper-g/n	A special case of the Hyper-g prior where u = g/n and u ~ Beta(1, alpha/2).	Liang, Paulo, Molina, Clyde and Berger (2007)
8	Hyper-g Laplace Approximation	A Laplace approximation to the Hyper-g priors	Wood and Butler (2002)
9	Zellner Siow	Zellner-Siow priors represented as a mixture of g priors with an Inv-Gamma(1/2, n/2):	Zellner and Siow (1980)

The Bayesian Information Criterior (BIC) prior is a reference prior based upon an approximation of the log marginal likelihood of parameter value \(\theta\) using the Bayesian information criterion.
\[log Pr(\theta|M_k) \approx c - 1/2BIC_k\] where:
\[BIC_k = nlog(1-R^2_k) + p_klog(n)\]
The \(R^2\) and \(p_k\) variables are the coefficient of determination and the number of regressors for model \(M_k\), respectively, \(n\) is the number of observations, and \(c\) is a constant that does not vary across models and so cancels in the model averaging. This prior is typically flat where the likelihood is large and contains the same amount of information that would be contained in a typical single observation (Wasserman and Kass 1996). The Akaike information criterion (AIC) prior is the same as the BIC prior above, except AIC is used to approximate the likelihood of the data given a model \(M_k\). The remaining priors are based upon the multivariate normal-gamma conjugate prior where the conditional distribution \(X|\theta \sim N(\mu,1/\lambda\sigma^2)\) and \(\beta \sigma^2\sim N(\beta_0, g\sigma^2S^{-1}_{XX})\), where the scaled variance and covariances are obtained from the ordinary least squares (OLS). This reduces prior elicitation down to two components, the prior mean of \(\beta_0\), taken to be 0, and the scalar g. Empirical Bayes Global Prior(EP-G) uses an EM algorithm to find a common or global estimate of g, averaged over all models (M. Clyde 2017). Empirical Bayes Local Prior(EP-L) uses the MLE of g from the marginal likelihood within each model (M. Clyde 2017). The Zellner’s g-prior is a multivariate normal-gamma conjugate prior where \(\beta \sigma^2\sim N(\beta_0, g\sigma^2S^{-1}_{XX})\), \(g = n\), and the scaled variance and covariances are obtained from the ordinary least squares (OLS).Hyper-g is a class of g-priors based upon continuous proper hyperprior f(g), giving a mixture of g-priors where the prior on g/(1+g) is a Beta(1, alpha/2) (M. Clyde 2017). Hyper-g-n is a special case of the Hyper-g prior where u = g/n and u ~ Beta(1, alpha/2), to provide consistency when the null model is true (M. Clyde 2017). Hyper-g Laplace is the same as the Hyper-g prior, but uses Laplace approximation to integrate over the prior on g. Lastly, the Zellner-Siow prior places a gamma prior on n/g with G(shape = 1/2, scale = 1/2).

2.3 Model Priors

For this experiment, the uniform distribution that assigns equal prior probability to all models was used such that \(Pr(M_k) = 1 / K\) for each \(k\) (A. E. Raftery 1988).

---

3.1 Data Sources

Launched in October 1990 by Col Needham, the Internet Movie Database (abbreviated IMDb) is an online database of film information, audience and critics ratings, plot summaries and reviews. As of November 2017, the site contained over 4.6 million titles, 8.2 million personalities, and hosts 80 million registered users (Needham 1990). Rotten Tomatoes.com, so named from the practice of audiences throwing rotten tomatoes when disapproving of a poor stage performance, was launched officially in April 2000 by Berkeley student, Senh Duong. It provides audience and critics ratings for some 26 million users worldwide (Flixter 2017). Founded in 1999, BoxOfficeMojo tracks box office information and publishes the data on its website (Fritz 2008).

3.2 Generalizability & Causality

The data were randomly sampled from available IMDb and Rotten Tomatoes website apis, and so the inference should be generalizable to the population. Since this was an observational study, random assignment was not performed, as such causality is not indicated by this analysis.

---

4.1 Univariate Analysis

4.2 Bivariate Analysis

Next, the relationships between the independent variables and audience scores are studied. The analysis continues with an exploration of the categorical variables vis-a-vis audience score, then an examination of the quantitative variables and the dependent variable.

4.3 Association and Correlation

Next, the significance and strength of associations between categorical variables were examined. Pairwise chi-squared and association tests were conducted to reveal the significance (p.value) and the strength, Cramer’s V (Hoel, Wolfowitz, and Cramer 1947) of each association. The chi-square results summarized in Table 20 reveal several associations that could present as collinearity issues for regression. Focusing on those regressors most highly associated with audience score, the degree of association among Academy awarded films was significant. There was also a strong association between films that won Best Picture and those that were nominated. Both significant and strong associations were observed among films with theatrical releases in the Oscar and Summer seasons.

Table 20: Chi-squared test of independence between categorical variables. Significant (\(\alpha = 0.05\)) differences between observed and expected differences under the null hypothesis are highlighted in bold text.

Terms	Feature Film	Drama	R Rating	Year	Oscar Season	Summer Season	Best Pic Nom	Best Pic Win	Best Actor	Best Actress	Best Director	Top 200
Feature Film	NA	0	0	0.001	0.468	0.243	0.293	0.906	0.032	0.044	0.077	0.476
Drama	0	NA	0	0.165	0.206	0.124	0.007	0.347	0.013	0.001	0.161	0.793
R Rating	0	0	NA	0.096	1	0.72	1	1	0.91	0.923	0.234	0.033
Year	0.001	0.165	0.096	NA	0.175	0.842	0.637	0.699	0.572	0.755	0.069	0.497
Oscar Season	0.468	0.206	1	0.175	NA	0	0	0.226	0.001	0.064	0.041	0.075
Summer Season	0.243	0.124	0.72	0.842	0	NA	0.098	1	0.045	0.729	0.813	0.701
Best Pic Nom	0.293	0.007	1	0.637	0	0.098	NA	0	0	0	0	0.155
Best Pic Win	0.906	0.347	1	0.699	0.226	1	0	NA	0.596	0.001	0	0.395
Best Actor	0.032	0.013	0.91	0.572	0.001	0.045	0	0.596	NA	0.001	0.017	0.32
Best Actress	0.044	0.001	0.923	0.755	0.064	0.729	0	0.001	0.001	NA	0.057	0.123
Best Director	0.077	0.161	0.234	0.069	0.041	0.813	0	0	0.017	0.057	NA	0.601
Top 200	0.476	0.793	0.033	0.497	0.075	0.701	0.155	0.395	0.32	0.123	0.601	NA

Table 21: Cramer’s V measure of association between categorical variables. Associations with a medium strength (Cramer’s V > .3) are highlighted in bold text.

Terms	Feature Film	Drama	R Rating	Year	Oscar Season	Summer Season	Best Pic Nom	Best Pic Win	Best Actor	Best Actress	Best Director	Top 200
Feature Film	NA	0.283	0.205	0.346	0.035	0.052	0.057	0.032	0.092	0.088	0.081	0.047
Drama	0.283	NA	0.155	0.284	0.053	0.064	0.115	0.052	0.103	0.138	0.061	0.021
R Rating	0.205	0.155	NA	0.293	0.003	0.017	0.002	0.014	0.009	0.009	0.053	0.094
Year	0.346	0.284	0.293	NA	0.283	0.229	0.247	0.242	0.251	0.237	0.298	0.256
Oscar Season	0.035	0.053	0.003	0.283	NA	0.443	0.179	0.064	0.133	0.078	0.087	0.081
Summer Season	0.052	0.064	0.017	0.229	0.443	NA	0.074	0.008	0.084	0.019	0.016	0.026
Best Pic Nom	0.057	0.115	0.002	0.247	0.179	0.074	NA	0.475	0.166	0.207	0.155	0.084
Best Pic Win	0.032	0.052	0.014	0.242	0.064	0.008	0.475	NA	0.042	0.154	0.332	0.083
Best Actor	0.092	0.103	0.009	0.251	0.133	0.084	0.166	0.042	NA	0.138	0.103	0.054
Best Actress	0.088	0.138	0.009	0.237	0.078	0.019	0.207	0.154	0.138	NA	0.085	0.077
Best Director	0.081	0.061	0.053	0.298	0.087	0.016	0.155	0.332	0.103	0.085	NA	0.041
Top 200	0.047	0.021	0.094	0.256	0.081	0.026	0.084	0.083	0.054	0.077	0.041	NA

As shown in Figure 35, the correlations among the quantitative variables did not surprise. As expected, a moderate correlation between critics scores and IMDb rating was observed.

                        data.imdb_rating data.critics_score data.runtime data.imdb_num_votes_log
data.imdb_rating               1.0000000          0.7646853    0.2643424               0.2057645
data.critics_score             0.7646853          1.0000000    0.1701741               0.0727904
data.runtime                   0.2643424          0.1701741    1.0000000               0.2902915
data.imdb_num_votes_log        0.2057645          0.0727904    0.2902915               1.0000000

Figure 35: Correlations among quantitative predictors

4.4 EDA Summary

Whereas acknowledgements of individual achievement from the Academy had no statistically significant correlation with audience scores, films that were nominated for, or won Best Picture, were associated with higher audience scores to a statistically significant degree. It would appear that audiences prefer teamwork. Non-feature films were also associated with higher audience scores. As one might expect, high positive correlations were extant between critics scores, IMDb ratings and audience scores. Weak positive correlations between IMDb number of votes and runtime were observed and finally, audience scores saw a slight, but statistically significant downward trend over the time period observed. The associations between the qualitative and quantitative variables revealed potential collinearity among the Oscar award variables. There was a moderate to strong correlation between critics scores and IMDb ratings.

---

5.1 Bayesian Model Averaging

Bayesian model averaging was conducted on the full data of 645 observations.

5.2 Model Performance

Now, the predictive performance of competing default priors are compared on the basis of mean squared error (MSE) on hold-out samples. Predictions were rendered using the BAS package (M. Clyde 2017), for four model estimators: (1) the BMA model, (2) the best predictive model (BPM), (3) the highest probability model (HPM), and (4) the median probability model (MPM) estimators. Predictions for each of the nine priors and the four model estimators were rendered for a total of 36 predictions. The movie data set was randomly split into a training set, \(D_{train}\), (80% of observations) which was used to train the model, and a test set, \(D_{test}\), (20% of observations) which was used to assess the quality of the resulting predictive distributions. The mean MSE was computed for each prior and estimator, for a total of 36 models. This analysis was repeated for the 36 models, 400 times for 400 different random splits. The average MSE of predictions for each model was computed as follows: \[1/j * 1/n_{test}\displaystyle\sum_{j = 1}^{t}\displaystyle\sum_{i = 1}^{n_{test}^{t}}(y_{test} - \hat{y}_{test})^2\] where:
\(t\) is the total number of trials
\(n_{test}^{t}\) is the total number of observations in in \(D_{test}^j\) \(y_{test}\) is the observed audience score for an observation in \(D_{test}^j\)
\(\hat{y}_{test}\) is the predicted audience score for an observation in \(D_{test}\)

Table 23 shows the predictive performance of the nine parameter priors and four estimators, in conjunction with the uniform model priors a evaluated by the MSE.

Table 23: Parameter priors and predictive performance by prior and estimator (movie dataset, model prior: uniform); 100 subsamples

Prior	BMA	BPM	HPM	MPM
Bayesian Information Criteria (BIC)	102.584	103.262	103.357	101.924
Akaike Information Criterion (AIC)	101.634	102.906	103.001	99.878
Empirical Bayes (Global)	102.586	103.292	103.404	101.935
Empirical Bayes (Local)	102.57	103.253	103.543	101.954
Zellner’s g-prior	103.116	104.355	105.179	101.683
Hyper-g	102.566	103.255	103.503	101.944
Hyper-g Laplace	102.562	103.252	103.473	101.934
Hyper-g-n	102.653	103.405	103.641	102.141
Zellner-Siow (NULL)	102.59	103.279	103.47	101.972
Note:
The five lowest MSE scores are in bold font. The lowest MSE is in red font.

The top five scores are highlighted in bold and the lowest average MSE is noted in red font. The Akaike Information Criterion (AIC) prior using the MPM estimator outperformed the other 35 models; however not decisively. To ascertain the significance of the differences in mean MSE among the models, t-tests of the MSE means were conducted at an \(\alpha = .05\) significance level. Table 24 reports the top four models in order of ascending mean MSE and the p.value for the probability that \(\mu_m = \mu_{best}\), where \(\mu_m\) is the mean MSE for model \(m\) and \(\mu_{best}\) is the lowest MSE among all models and estimators. As indicated by the p.values, the differences in mean MSE were not significant. Further, the box plot of the distribution of MSE scores in Figure 39, reveals significant overlap in the distributions of MSE. Though other experiments showed differences which were significant, this plot supports the notion that the each of the four models should be given some consideration.

Table 24: Mean MSE by prior and estimator for top four models with p.values for the probability that \(\mu_m = \mu_{best}\), where \(\mu_m\) is the mean MSE for model \(m\) and \(\mu_{best}\) is the lowest MSE among all models and estimators.

Prior	Estimator	Mean MSE	p.value
Akaike Information Criterion (AIC)	MPM	99.88	1.00
Akaike Information Criterion (AIC)	BMA	101.63	0.03
Zellner’s g-prior	MPM	101.68	0.02
Bayesian Information Criteria (BIC)	MPM	101.92	0.01

Figure 39: Distribution of MSE among top 4 models

5.3 Parameter Estimates

As indicated in Table 25, there was some variation among the selected predictors as well as their estimates. Markedly, all four models produced precisely the same estimate for the intercept, although with slightly different levels of uncertainty. The single greatest positive influencer of audience scores, IMDb ratings would generate between 14-15 audience score points for each point on the IMDb rating scale. All estimates had similar variations with \(SD \approx 0.58\). Like IMDB rating, critics score was a positive factor in all four models, but with far less influence on audience scores than was IMDb rating. A point on the critics score scale would result in an estimated 0.7 points on the audience score scale. There was a slight negative association between runtimes and audience scores (approximate 0.05 point drop for each minute of run time) in the AIC (MPM and the g-prior (MPM) models. Run time wasn’t a factor in the EB-Global (MPM) and Hyper-G-Laplace (MPM) models. An MPAA R rating wasn’t necessarily good for a film’s audience score. The AIC (MPM) and g-prior (MPM) models associated an R rating with a drop of approximately 1.5 points in the audience score. The Oscar performance of the films were relatively significant predictors in the AIC(MPM) model, but the devil is in the details. Only Best Picture nominations were positively associated with audience scores, with a relatively high estimate of 5 points; however, individual performance was not rewarded. Best actor and best actress winners were negatively associated with audience scores. It would appear that audiences prefer ensembles.

Table 25: Coefficient Estimates by Prior and Estimator

Terms	Mean	SD	2.5%	97.5%
AIC (MPM)
Intercept	62.3907	0.3913	61.6222	63.1592
feature_filmyes	-4.8894	2.0606	-8.9358	-0.8430
dramayes	1.6979	0.8961	-0.0619	3.4576
runtime	-0.0662	0.0232	-0.1118	-0.0207
mpaa_rating_Ryes	-1.4881	0.8101	-3.0789	0.1026
thtr_rel_year	-0.1013	0.0397	-0.1793	-0.0233
imdb_rating	14.2430	0.6231	13.0194	15.4665
critics_score	0.0624	0.0219	0.0193	0.1055
best_pic_nomyes	4.1994	2.3364	-0.3886	8.7874
best_actor_winyes	-1.8900	1.1683	-4.1841	0.4042
best_actress_winyes	-2.2362	1.3050	-4.7990	0.3265
imdb_num_votes_log	0.6652	0.2192	0.2347	1.0957

Terms	Mean	SD	2.5%	97.5%
AIC (BMA)
Intercept	62.3907	0.3936	61.6342	63.1461
feature_filmyes	-3.0951	2.7581	-8.2560	0.0070
dramayes	0.7515	0.9893	-0.1756	3.0441
runtime	-0.0554	0.0289	-0.1000	0.0000
mpaa_rating_Ryes	-1.0027	0.9733	-2.8649	0.0006
thtr_rel_year	-0.0729	0.0534	-0.1625	0.0000
oscar_seasonyes	-0.3496	0.7349	-2.3544	0.4259
summer_seasonyes	0.2694	0.6540	-0.5826	2.0751
imdb_rating	14.5169	0.6935	13.2281	15.9119
critics_score	0.0644	0.0248	0.0160	0.1167
best_pic_nomyes	2.9877	2.9291	-0.0289	8.7413
best_pic_winyes	-0.5984	2.7371	-8.2414	4.3568
best_actor_winyes	-1.0102	1.2676	-3.7747	0.2247
best_actress_winyes	-1.2409	1.4562	-4.2942	0.1893
best_dir_winyes	-0.7120	1.3876	-4.2047	0.9884
top200_boxyes	0.2480	1.5317	-3.1588	4.1925
imdb_num_votes_log	0.4712	0.3087	0.0000	0.9858

Terms	Mean	SD	2.5%	97.5%
g-prior (MPM)
Intercept	62.3907	0.3943	61.6165	63.1649
runtime	-0.0688	0.0220	-0.1120	-0.0257
mpaa_rating_Ryes	-1.6788	0.7914	-3.2328	-0.1249
thtr_rel_year	-0.0676	0.0374	-0.1411	0.0059
imdb_rating	14.7885	0.5914	13.6271	15.9498
critics_score	0.0721	0.0218	0.0293	0.1149
imdb_num_votes_log	0.3787	0.1803	0.0248	0.7327

Terms	Mean	SD	2.5%	97.5%
BIC (MPM)
Intercept	62.3907	0.3981	61.6089	63.1725
imdb_rating	14.6058	0.5737	13.4792	15.7324
critics_score	0.0746	0.0218	0.0319	0.1174

5.4 Top Model Evaluations

Having explored prediction accuracy, and characterized the parameter estimates in terms of their effects on audience score, the focus turned to determining the quality of those model inferences. Specifically, the distribution of errors vis-a-vis predictions were evaluated for homoscedacticity. Figure 40, presented very similar distributions of errors vis-a-vis fitted values for each of the nine priors. A greater concentration of residuals was observed at the higher audience scores; whereas the residuals for lower scores were more dispersed. Also, the quality of the predictions tended to be positively associated with the quantity of observations at any particular range of audience scores.

6.1 Prediction Data

The five movies selected for prediction (Table 26) were so chosen to present a range of values for audience score and predictor values across the parameter space. Audience scores ranged from 34 to 87. The number of IMDb votes varied from approximately 8000 votes for Synchronicity, to over 460,000 for Suicide Squad. Similarly critics scores stretched from a low 26 points to a near perfect 98. Results at the Academy varied significantly among the chosen films in order to provide a reasonably diverse sampling.

Table 26: Movies selected for prediction

Title	Feature	Drama	Run	R	Year	Oscar	Summer	IMDb Rating	Critics	BP Nom	BP Win	Actor	Actress	Dir	Top	# Votes	Audience
Split	yes	no	117	no	2016	no	no	7.3	75	no	no	no	no	no	no	252594	78
Moonlight	yes	yes	111	yes	2016	yes	no	7.5	98	yes	yes	no	no	no	no	184405	79
Rogue One: A Star Wars Story	yes	no	133	no	2016	yes	no	7.8	85	no	no	no	no	no	yes	411283	87
Syncronicity	yes	yes	101	yes	2016	no	no	5.5	46	no	no	no	no	no	no	8176	34
Suicide Squad	yes	no	130	no	2016	no	yes	6.1	26	no	no	no	no	no	yes	466363	60

6.2 Prediction Results

Table 27: summarizes the predictions, true values, and mean squared errors for each movie and film. Mean MSE across the four models and five films ranged between 6.4 and 6.7., where the Zellner’s g-prior performed best. All models under-predicted audience scores for Split, a ‘middle-of-the-pack’ film with no Academy awards, with critics scores and IMDB ratings in the 70th percentile. By contrast, all models over-predicted audience scores for Moonlight, a film enjoyed by the critics and the Academy, but had the 2nd lowest number of votes. The audience score for Rogue One was a blockbuster film with over 400,000 votes. The largest variance among all the models was the significant over prediction for Synchronicity, a relatively unknown outlier, as indicated by the number of votes, which was panned by the critics. Predictions for Suicide Squad, the most frequently rated of the five, were significantly lower than the true audience scores.

Table 27: Prediction Results

Film	AIC	EB-global	g-prior	hyper-g-laplace	Audience Score
Split	75.2	75.4	75.8	75.4	78
Moonlight	83.4	80.1	79.2	80.1	79
Rogue One: A Star Wars Story	82.5	83.5	83.2	83.5	87
Syncronicity	45.7	47	46.2	47	34
Suicide Squad	53.4	54.3	53.8	54.3	60
MSE	6.7	6.7	6.4	6.7

The thread that goes through all the predictions involves estimate for the IMDb (log) number of votes parameter. Models over predicted audience scores for films with the highest number of votes and penalized those with few votes. A log transformation was applied to the this variable to address significant right skew. Future studies might experiment with more sophisticated transformations or higher order polynomials to more precisely characterize the relationship between votes and audience scores,

7.1 Default Priors and Prediction Models

7.2 Default Priors and Predictive Performance

This paper documents the exploration of default priors and their effect on predictive accuracy in Bayesian model averaging. Predictive accuracy was evaluated for BIC and AIC model selection techniques, Zellner’s g, local and global empirical Bayes, as well the g-prior mixtures, Zellner Siow, Hyper g and Hyper-g/n priors. BMA, highest probability, best predictive, and median predictive models were fit for each prior, for a model space of 36 separate models. A data set of 645 observations was randomly split into a training (80%) set, which was used to develop the models, and a test (20%) set which was used to evaluate the predictive accuracy of each model on a mean squared error basis. This process was repeated for 400 trials, MSE was recorded and averages were computed for each model and estimator. The AIC prior generated predictions with the lowest mean MSE of the 36 models during this experiment, followed by Zellner’s g-prior, global empirical Bayes and Hyper-g using Laplace approximation. In each case, the median probability models estimators outperformed BMA, highest probability models, and the best predictive model. That said, the differences in mean MSE were not significant enough to make any conclusive statements with respect to the relative performance accuracy of these four priors.

7.3 Default Priors and Prediction Models

---